18. Data Quality

Requirements tell engineers exactly how to build a data pipeline

Requirements are how we can set and measure quality

Requirements allow both engineering and non-engineering roles to agree on the high-level method for preparing the output.

Requirements tell engineers what the output of their data pipelines should be

Use a Service Level Agreement

Use a schedule

Use a start date

Use an end date

Data must be accurate to some margin of error

Data must arrive within a given timeframe from the start of execution

Pipelines must run on a particular Schedule

Data must not contain any sensitive information

Data must be of a certain size

Data must be of a certain size

Data must be accurate to some margin of error

Data must arrive within a given timeframe from the start of execution

Pipelines must run on a particular Schedule

Data must not contain any sensitive information